Experimental study I Videosurgery 

In vitro evaluation of stereoscopic liver surface reconstruction 



Dominik Spinayk', Adam Karwan^ Jaroslaw Zylkowski', Tadeusz Wrdblewski* 

^Institute of Biomedical Engineering, Silesian University of Teclinology, Giiwice, Poland 

^Institute of Informatics, Silesian University of Technology, Giiwice, Poland 

^Second Department of Clinical Radiology, Medical University of Warsaw, Poland 

^Chair and Department of General, Transplant and Liver Surgery, Medical University of Warsaw, Poland 

Videosurgery Miniinv 2013; 8 (1): 80-85 
DOI: 10.5114/wiitm.2011.32809 

Abstract 

Introduction: Tracking abdominal motion of organs is an important factor in image-guided navigation systems. The 
paper presents the evaluation methodology of a practical approach to measure liver motion, both respiratory and 

laparoscopic, with a tool guided in the operating room. 

Aim: Evaluation of the methodology of a practical approach to measure liver motion, both respiratory and laparo- 
scopic, with a tool guided in the operating room. 

Material and methods: The presented evaluation method is based on standard operating room equipment, i.e. 
laparoscopic cameras. We decided to use two rigid cameras to gain stereo in order to reconstruct characteristic 
points by triangulation. Our research aim was to survey the impact of three parameters on reconstruction accu- 
racy: the number of calibration points, the imprecision of camera assembly, and the difference in resolution of 
images. 

Results: Three calibration chessboard configurations were tested. The reconstructed landmark positions and residual 
mean square errors were presented in three phantom poses: the reference position, translated position and rotated 
position. 

Conclusions: The presented approach is a development of the previous work. Our research proved the importance of 
a rigid stereo camera system and the use of high definition image resolution for both stages, namely calibration and 
reconstruction. 
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Introduction 

Tracking abdominal motion of organs is an im- 
portant factor in image-guided navigation systems. 
The paper presents the evaluation methodology 
of a practical approach to measure liver motion, 
both respiratory and laparoscopic, with a tool guid- 
ed in the operating room. The evaluation was bas- 
ed on the methodology described in the previous 
paper [1]. 



Aim 

The proposed evaluation methodology can be 
divided into three stages: configuration of the stereo 
camera, camera calibration, solving of the correspon- 
dence problem. Our research aim was to survey the 
impact of the following three parameters on recon- 
struction accuracy: the number of calibration points, 
imprecision of monocular camera assembly, standard 
and high resolution images. 
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Material and methods 

Reconstruction of point X in 3D space, called the 
world coordinate system, requires solving the corre- 
spondence problem from two monocular cameras [2]: 

X = x{Pl, Pp, Xl, xp), 

where t denotes the triangulation algorithm, P^, Pp 
are projection matrices of the left and right monocu- 
lar camera, and Xi^, Xp are coordinates of correspon- 
dence point in the left and right images. 

The first step was to set up the equipment and 
conduct the calibration procedure. In our approach 
Tsai's algorithm was selected as a calibration method 
[3, 4]. We tested Tsai's method with different configu- 
rations of the calibration chessboards and different 
numbers of the calibration points. Both cameras should 
remain in the same relative position after the cali- 
bration procedure. The calibration procedure requires 
a few pose acquisitions of the calibration grid from 
both cameras, which is difficult during a laparoscopic 
procedure. It is impossible to use standard laparoscop- 
ic cameras to grab required poses directly in a patient 
position in the operating room. After taking out the 
cameras from trocars, cameras are reassembled in the 
same relative position to each other. Inaccuracy of this 
process influences projection matrices P, whose enti- 
ties are decomposed as follows [2]: 

P = l<[R\t], 

where: K - calibration matrix with internal camera 
parameters, R - rotation matrix in the world coordi- 



nate system, f- translation vector in the world coor- 
dinate system. 

After calibration of a stereo camera and setting up 
the equipment (Figure 1) we focused on localization 
of correspondence points in the pair of monocular 
camera images. 

The accurate reconstruction from 2 monocular 
videos require two types of adequacy: 

• frame correspondence in time dimension, 

• point correspondence in the image from both cam- 
eras. 

The best solution is obtained by hardware syn- 
chronization. We rejected this approach because the 
assumption is to use standard laparoscopic cameras 
which do not make it possible. The proposed solution 
is based on time stamps of a video frame sequence, 
using time stamps from the AVI file format and the 
maximum time-shift threshold approach. The evalua- 
tion was done on exactly the same equipment that is 
used in the operating room: medical video processor 
Olympus Exera II CV-180 with constant value frame 
per second parameter, which defines our maximum 
time-shift threshold. Pairs of frames whose time-shift 
is bigger than threshold are rejected. 

In the presented approach [1] two well-known 
methods for solving the correspondence problem 
were taken into account: 

• RANdom SAmple Consensus (RANSAC) [3], 

• Lucas-Kanade [5]. 

RANSAC and Lucas-Kanade require landmarks to 
be obtained in the pair of corresponding images. For 
evaluation of possible triangulation accuracy, we 
used artificial markers attached to an abdominal liv- 
er phantom (Photo 1). The defined marker pattern. 



Liver phantom 



Figure 1. Setup diagram 
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Photo L Liver pliantom with attached landmarl<s 
and its image positions 




Photo 2. Calibration chessboards' configuration 
with marked axes of world coordinate system 

as a characteristic configuration of white and black 
boundaries of the image areas, was applied to locate 
markers' positions. Selecting correspondence points 
in pairs in both images was performed manually. 

Results 

The selected configuration of the calibration chess- 
board in our approach is presented below (Photo 2). 



Three calibration chessboard configurations were 
tested: both monocular cameras were calibrated on 
the left chessboard, both monocular cameras were 
calibrated on the right chessboard, the left camera 
was calibrated on the right chessboard and the right 
camera on the left one. 

Generally we tested triangulation results on two 
data sets: left and right chessboard corners, liver 
phantom assigned landmarks. 

The first data set allows an easy evaluation of 
results because we know the reference position of 
chessboards which define the world coordinate 
system (Photo 2). We use 30 mm chessboards with 
81 corners. For the above-mentioned three calibration 
configurations the origin of the world coordinate sys- 
tem was fixed in the right corner of the left calibration 
chessboard (Photo 2). Moreover, we did the tested tri- 
angulation in SD/HD with 27, 45 or 81 points (which 
means: 3 upper rows of chessboard corners, 5 upper 
rows of chessboard corner rows and all chessboard 
corners respectively). The average residual mean 
square error (RMSE) in millimetres is presented below 
(Table I). There is anomaly in RMSE for 45 calibration 
points in HD resolution that is a little greater than for 
27. However, this exception does not affect the ten- 
dency that with more calibration points we obtain 
higher accuracy (less RMSE). 

The reconstruction positions of liver phantom 
markers were tested in three positions in the high 
resolution mode: reference position, translated posi- 
tion, rotated position. 

The reconstructed position and the average RMSE 
in millimetres are presented below (Tables II, III). The 
reconstructed liver phantom landmarks in three posi- 
tions are presented (Photo 3). 



Table I. Average residual mean square error of chessboard corner triangulation 



Camera configuration 


Number 
of calibration 
points 


Left chessboard 


Right chessboard 




Both chessboards 


Camera resolution 


Camera resolution 




Camera resolution 


Left Right 


Left Right 




Left 


Right 
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HD SD HD SD 


HD 


SD 
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1.78 


3.28 


3.43 
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1.80 


2.83 


3.34 


81 
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0.16 0.21 


1.61 


2.09 


1.19 3.16 
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Table II. Phantom landmarks' reconstructed positions 





Reference position 






Translated position 






Rotated position 




A 


Y 


z 




y 

A 


Y 


z 


r\IVlj 


Y 
A 


Y 


z 


r\IVl J 


—71 ^ Q7 


31.57 


-143.12 


^ 70 


JOO.OU 


35.22 


-164.98 


A OR 


— ^RA R^ 
30D.03 


38.86 


-164.06 


9 Q'; 


—HQ 91 


41.81 


-107.13 




— ZL1 ^ ZL9 
^13. HZ 


43.77 


-138.69 


A 1ZL 


— ZLII RO 


50.05 


-135.08 


9 11 
Z.ii 


ZZJ.Zi 


51.85 


-144.67 


3.00 




53.62 


-172.18 


A n^ 

D.U3 


— QQZL 
3i'H-.3 J 


59.05 


-169.41 


^ OZL 


— ?^ZL 91 


66.25 


-130.41 


^ 7^ 
J. / o 




67.40 


-161.56 


j.yj 


— 4.rm 1ZL 

HU3.IH 


76.30 


-155.43 


9 R9 
z.oz 


— iQ^ m 


24.21 


-199.28 




J JU. JO 


27.15 


-213.25 


A n^ 


— ^R 
3 J3.30 


27.47 


-214.23 


H.J J 


—1 Qfi m 


16.05 


-214.37 




3 J0.30 


19.68 


-223.04 


A ^1 


— ^ZLI ^9 
3HI. JZ 


20.16 


-223.99 


A A^ 


— IRO ID 


4.45 


-227.70 


^ 7^^ 
J. / o 


—^9^ 7n 

3Z3. / U 


7.81 


-231.55 


^ A^ 


—^97 Q7 
jA/.y/ 


7.29 


-232.78 


ZL 7ZL 
H. /H 


—1 ^(^ 0^ 


-6.80 


-234.26 


^ ZLf~) 


— ^flQ 99 
3U^.ZZ 


-3.44 


-234.71 


^ A9 
J.DZ 


—^1 ^ 9^ 
31 J.Z3 


-4.49 


-236.34 


H.^3 


—9m 11 


49.48 


-177.58 




— ^7A 1 9 
3 / D.IZ 


52.86 


-197.16 


A 9A 
D.ZO 


— ^7ZL ^9 
3 /H-.3Z 


56.72 


-195.63 


A OZL 


ZUO. / D 


46.16 


-197.61 


ZL 9^ 


— 9^ 
3D3.Z J 


48.77 


-214.40 


A ZL1 
D.Hi 


— ^Afl R9 
3DU.OZ 


52.42 


-212.99 


ZL 


— 1QQ 9A 


40.25 


-211.55 


j.yj) 


— ^zlQ 77 


42.49 


-225.13 


^ 79 
J. / z 


— ^ZLQ 07 


43.83 


-224.76 


ZL R7 


—178 f^ZL 


24.52 


-237.51 


A 9A 


—^91 79 
3Zi. / Z 


27.04 


-242.80 


J.3y 


—^9^ 7Q 
3Z3. / 7 


26.39 


-243.32 


^ 1 R 

J.iO 


—zzz. 44 


66.60 


-177.36 


3.93 


— 3ol./6 


69.51 


-199.30 


^ 3 O 

6.28 


—378.14 


74.86 


-196.73 


4.25 


-212.14 


66.13 


-195.56 


3.82 


-368.51 


68.19 


-215.55 


6.27 


-365.49 


72.32 


-213.88 


4.68 


-206.10 


61.79 


-212.20 


4.22 


-356.32 


64.34 


-227.76 


6.23 


-354.00 


67.30 


-226.73 


5.03 


-194.04 


54.02 


-227.88 


3.98 


-339.67 


56.68 


-239.06 


6.06 


-338.45 


58.30 


-238.62 


5.27 


-184.74 


45.09 


-239.63 


4.43 


-327.30 


47.25 


-247.20 


6.38 


-327.71 


48.06 


-247.23 


5.47 


-222.32 


85.05 


-172.96 


3.97 


-385.12 


86.50 


-197.06 


6.19 


-380.12 


93.51 


-193.55 


4.19 


-207.98 


81.98 


-210.78 


4.49 


-359.53 


83.68 


-228.28 


6.57 


-355.47 


88.11 


-226.58 


5.51 


-198.54 


75.41 


-226.17 


4.04 


-345.17 


77.61 


-239.40 


6.10 


-341.77 


80.77 


-238.29 


5.42 



Table III. Average residual mean square error of 
phantom landmarl<s' triangulation 

Phantom position - average RMSE 

Reference position Translated position Rotated position 

3.88 6.07 4.21 



The comparison of results (Table I) indicates the 
advantage of the calibration in High Definition Mode 
over the calibration in Standard Definition. The sug- 
gested number of chessboard calibration points for 
Tsai's method is more than 30 [2]. The performed 
tests highlight the decrease of RMSE for a higher 
number of calibration points, with one exception - 
discussed in the previous paragraph. More accurate 




Photo 3. The reconstructed liver phantom land- 
marks in 3 positions: reference position (white), 
translated position (red), rotated position (green) 



results were obtained for calibration with one chess- 
board (left or right), where average RMSE is less that 
0.5 mm, compared with calibrating cameras on dif- 
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ferent chessboards (in the worst case average RMSE 
is 1.61). 

Applying two chessboards for calibration does not 
cause different results in RMSE for any of the chess- 
board corners. 

Discussion 

As regards the calibration chessboards setup by 
using one chessboard we assume that image noise is 
identically distributed in the observed coordinates, 
which generally is not true. In order to minimize tri- 
angulation RMSE in the whole reconstruction volume, 
it is advisable to select the set of calibration points 
from the whole volume. Since it is difficult to obtain, 
one calibration chessboard is used in practice [3]. 
There are some descriptions of research using a set 
of calibration grids instead of one grid [6]. We tested 
both approaches. The achieved results have less 
RMSE in the case of one chessboard. The impossibili- 
ty to calibrate rigid configuration of two monocular 
laparoscopic cameras in the operating room (OR) 
patient position required reassembly of the camera 
setup, which caused RMSE increase in the case of 
two perpendicular chessboards. Taking OR circum- 
stances into account, one calibration chessboard 
seems to be enough. 

Tsai's algorithm is a well-known solution for a ca- 
mera calibration problem. It requires at least 11 cali- 
bration points [2], but normally between 20 and 60 
are used. We applied two perpendicular, 30 mm cali- 
bration chessboards with 81 corners and tested 
21/45/81 calibration points in both high and standard 
definition modes. The high camera resolution mode 
has less RMSE than the standard one. This is espe- 
cially important if one calibration grid is used to cali- 
brate both cameras (differences in RMSE between 
the high and standard mode are significant). The 
experiments proved the important impact of image 
resolution on triangulation accuracy 

The triangulation approach for surface recon- 
struction requires solving the correspondence prob- 
lem. Two well-known approaches, RANSAC [5] and 
Lucas-Kanade [7], require localization of landmarks in 
pairs of corresponding images. For possible evalua- 
tion of triangulation accuracy we use artificial mark- 
ers attached to the abdominal liver phantom. Our 
approach, which is based on a defined marker pat- 
tern, as a characteristic configuration of white and 
black boundaries of the image area, could not find all 



instances of markers (Photo 1). The reason for the 
omitted markers was a large difference from the 
defined pattern. To reconstruct a more complex liver 
surface we plan to use a Time of Flight camera for 
open surgery [8]. 

The sensitivity of the triangulation approach is 
a relevant issue in investigation on 3D reconstruction 
[9]. Measurements of differences in the reconstruc- 
tion position by rotation of one out of two cameras at 
a few degrees in the operating room may cause a sig- 
nificant difference in reconstruction position of the 
liver surface (Photo 3). This is a challenge in the la- 
paroscopic case where generally it is impossible to 
calibrate cameras in the same position as used for tri- 
angulation (patient position). The best solution for 
this approach is to use one stereo laparoscopic cam- 
era. There is one available solution which is a stereo 
camera in laparoscopic teleoperated robot-assisted 
surgery, da Vinci, but the price and long learning 
curve for an operating team are crucial obstacles for 
common use. Nowadays there is a lack of cheaper 
and implemented medical solutions from competi- 
tive companies. 
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